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ABSTRACT 



This feasibility study explored the automated data 
collection, scoring, and reporting of children's complex problem-solving 
processes and performance in Web-based information-rich environments. Problem 
solving was studied using realistic problems in realistic contexts demanding 
multiple cognitive processes in the domain of environmental science. 
Sixty-nine middle school and high school students completed pretest and 
posttest concept maps, a relevant bookmarking measure, and a metacognitive 
survey. Process data was collected using computer trace data. During posttest 
map construction students had access to a simulated World Wide Web 
environment. The relevant bookmarks students used were web documents, 
specific universal resource locators, that helped students make at least one 
meaningful link between two or more concepts. Students were trained in the 
computer-based concept mapping software and then were given 20 minutes to 
construct a closed concept map about environmental science. After a tutorial 
in the bookmark application and the simulated Web environment, students were 
instructed to search the simulated Web for 40 minutes to help them improve 
their maps. Data are not complete for all participants due to computer 
crashes and data log management problems. Student performance from pretest to 
posttest for concept mapping scores did improve, but more exploratory 
information seeking behavior did not predict higher scores. Extracting 
relevant information from the Web resulted in higher final concept mapping 
scores, but high scores on metacognition did not predict high scores on the 
relevant bookmarks and final concept maps. There were no significant results 
for the use of feedback contributing to higher final concept mapping 
performance or a greater number of relevant bookmarks. (Contains 43 
references.) (SLD) 
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Feasibility of a Web-Based Assessment of Problem Solving 

In the feasibility study reported here, we discuss our first attempt at automated data 
collection, scoring, and reporting of children’s complex problem solving processes and 
performance in Web-based information rich environments. 

We situate this research within the CRESST model of families of cognitive 
learning. Those families include: collaboration, communication, content understanding, 
metacognition, and problem solving. In assessing student problem solving, we address the 
cognitive components of content understanding (i.e., the student's ability to understand 
important facts, procedures, concepts, and principles related to a content domain), 
metacognition (i.e., the student's self-assessment of awareness, planning, and deployment 
of cognitive strategies), and problem solving, (i.e., the cognitive process directed at 
achieving a goal when a solution method is not obvious to the problem solver (Chi & 
Glaser, 1985; Dorner, 1983; Jausovec; 1994; Holyoak, 1995; Howard, 1983; Mayer & 
Wittrock, 1996; Simon, 1973; Voss & Means, 1989). 

We argue, however, that this definition of problem solving lacks what is most 
crucial to problem solving — the processes behaviors that lead to or detract from successful 
performance. If a goal of assessment is to bridge the gap between testing and instruction, 
then assessments need to report not only the students final performance, but also what 
processes students need to improve on to bolster their performance. This instructional 
sensitivity has been an unrealized goal of both large scale and small scale performance 
assessment (Baxter, Elder & Glaser, 1996; Picus, 1996) 

Assessing Problem Solving in the Large 

We make the distinction here between problem solving in the “large” and problem 
solving in the “small”. By problem solving in the “large” we mean assessing students 
cognitive abilities when working on realistic problems in realistic contexts that demand that 
students activate multiple cognitive processes to grapple with multiple competing solutions 
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(Domer, 1983; Meacham & Emont, 1989; Voss, Tyler, & Yengo, 1983). These attributes 
differ greatly from problem solving in the “small” in which competing solutions and 
alternatives are reduced by presenting learners with well-defined one-to-two step problems 
(Young & McNeese, 1995). Problem solving in the “small” scenarios ignore the 
perceptual information regarding potential solutions that a more realistic context provides 
(Cognition and Technology Group at Vanderbilt, 1996; Young & McNeese, 1995; Voss & 
Post, 1988). 

Since our focus is on assessing problem solving in the “large,” we have employed a 
realistic problem solving context (i.e., an information rich Web-based environment) that we 
believe will be part of all students future learning. This context situates students in 
electronic information rich environments like the Internet or Intranets, presents students 

with an ill-defined problem or problems, and gives them access to computational tools that 

i S 

will help them reach problem resolution. We argue that problem solving skills in 
information rich environment include: information seeking and retrieval, extracting relevant 
information, simplifying information, and then organizing and using the information one 
finds, to further one’s understanding. 

A Hurdle 

As alluded to above, detailed reporting of students cognitive processes while 
engaged in learning activities has been a hurdle in attempts to link assessment with 
instruction (Baker, O’Neil, & Linn, 1993; Picus, 1996; Resnick, 1996). Even in small 
scale assessment, collecting and reporting the kinds of detailed qualitative analyses of 
students processes are tremendously time intensive and expensive (Glaser & Silver, 1994; 
Huberman & Miles, 1994; Vidich & Lyman, 1994). 

To combat the time intensive and monetary issues of using humans to collect and 
analyze process data, we had our problem solvers work in a computer networked 
environment. The benefits of networked environments are that they log and store 
automatically all the process behaviors of those engaged at the computer . Thus, using 
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networked technologies enabled us to overcome the hurdle of collecting, coding and 
scoring student process and performance data which in turn enabled us to report on both 
how the student performed, and the process behaviors the occurred during that 
performance. 

Problem Solving Processes 

Our ideas of problem solving processes are quite similar to those proposed over 30 
years ago by Jerome Bruner (1966). Bruner claimed that problem solving processes 
involve: (a) exploration, (b) extraction of relevant material, and then (c) simplification and 
(d) organization of that material. 

Exploration and Extraction . Researchers who study ill-structured problems — 
problems that require resolving a number of open constraints (Rietman, 1965), 
decomposing the more global problem into a set of well-structured problems (Simon, 

1973), making sense of vague goal criteria (Rittel & Webber, 1972), and managing 
“multiple goals” to satisfy several criteria (Domer, 1983) — have found that resolution of 
complex problems is dynamic. Often, important features of a problem reveal themselves 
only after one has delved into the task (Churchman, 1971; Bennett, 1993; Domer, 1983; 
Simon, 1973; Voss, Tyler & Yengo, 1983). Many researchers argue because of this fact 
that success on complex problems requires high degrees of exploratory behavior. These 
behaviors include: viewing ideas from a large number of perspectives, avoiding imposing a 
dominant central organization on the problem, and dealing with several dimensions and 
concepts within a multidimensional phenomena (Bates, 1986; Bruner, 1966; Feltovich, 
Spiro, Coulson, Feltovich, 1996). 

To uncover these exploratory processes, researchers that study problem solving 
have employed think aloud protocols of individuals while problem solving or interviewed 
individuals after their problem solving experience (Simon & Ericcson, 1993; Voss et. al, 
1983; Voss & Post). In our work, we have taken a different approach to measuring 
exploratory problem solving processes. Instead of conducting interviews, naturalistic 
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observation, or think aloud protocols, while students problem solve, we rely on server 
access logs to automatically record all information seeking and retrieval behaviors the 
student engages in. 

We measure exploratory process behaviors based on information science 
researchers findings that information seeking and retrieval is an exploratory problem 
solving process. A process which involves recognizing an information problem, defining 
that problem, formulating a query, browsing, examining results for relevant documents, 
and deciding when to stop (Kulthau, 1993; Marchionini, 1995). Furthermore, information 
seeking has been found to be iterative and dynamic (Bates, 1986; Bates, 1989). When 
users interact with previously unencountered information, additional aspects of the problem 
may start to reveal themselves, and what one thought was once relevant information may 
no longer be. The processes of defining and limiting information needs, formulating 
queries that fit the system, and then deciding what is relevant and why (Belkin, Oddy, & 
Brooks, 1982; Blair, 1996; Borgman, 1996; Borgman et al., 1995; Ellis, 1996; Moore, 
1995), all require reasoned problem solving behaviors. We argue that a Web environment 
that encourages electronic information seeking and retrieval can measure the complex 
problem solving processes of exploring information (i.e., browsing, analytic querying, 
scan and select behaviors), analyzing needs (i.e., determining what is relevant and why), 
and extracting relevant information. In simpler terms we measured exploratory problem 
solving processes that contributed to finding relevant information. 

Simplification and Organization. 

Those who seek information in electronic environments don’t always find what 
they are looking for. Further, even if students find information, that is no guarantee that 
they will be able to understand and use it. Along with exploring and extracting 
information, students also need to do something with the information they find. Using 
what one finds requires that the problem solver simplify and organize information so that it 
can be comprehended (Bruner, 1966). In our problem solving environment, we integrate a 
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Web-based information space with a set of tools (i.e., a concept mapper and relevant 
bookmarking measure) to help problem solvers use what they find. We had students solve 
the problem of developing an elaborated concept map. We could have had children write 
essays, develop outlines for oral reports, initiate research questions, design specifications 
for software, or work on a shared simulation. The combination of any ill-defined task with 
an electronic information space will involve the interaction between the person, the 
information, and the problem to be solved. Organizing and simplifying information from a 
Web environment such that it can be meaningfully used is critical. Organizing one’s own 
cognition by monitoring what one knows and does not know, when one should search, 
when one should read, when one should simplify, and when one should extract 
information is equally important. 

Doug Engelbart (1962) argued that if there is any one thing upon which intellectual 
growth depends, it would seem to be organization. If knowledge could be organized and 
easily accessible down to its beginnings, “we will have amplified the intelligence of the 
human by organizing his intellectual capabilities into higher levels of synergistic 
structuring” (Engelbart, 1962, p. 19). Without the ability to simplify and organize (a) 
information found in a Web environment, and (b) information accessed within one’s own 
cognition, problem solvers in electronic environments may very well flounder. The sheer 
volume of information available may lead children to aimlessly wander through cyberspace. 
Similarly, the massive amounts of information may overwhelm a problem solver such that 
he or she is unable to make use and sense of the information he or she comes across. 
Without the capacity to simplify and organize materials encountered, problem solvers 
solutions will resemble compilations of disjointed bits of information that evidence only 
surface level understanding. 
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Hypotheses 



The following hypotheses address how the processes of exploration, extraction, 
simplification, and organization contribute to student problem solving performance on 
concept mapping and information seeking tasks. 

(1) With the addition of an electronic information space, students performance from pre-to 
posttest concept mapping scores will improve. 

(2) More exploratory information seeking behaviors (i.e., browsing, scan and select, and 
analytic information seeking) will be predictive of higher scores on finding relevant 
information and higher scores on final concept mapping. 

(3) Extracting more relevant information (i.e., bookmarking) from the Web environment 
will predict higher final concept mapping scores. 

(4) High scores on metacognition will predict high scores on relevant bookmarks and final 
concept maps. 

(5) High scores on task simplification (i.e., focusing on bookmarking concepts one knows 
little about) \yill predict higher final concept mapping scores. 

(6) Students who organize and regulate their learning by accessing feedback on their 
concept mapping performance will score higher on final concept maps and finding relevant 
information than those who do not. 
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Methods 



Design 

This was an exploratory study that assessed students learning and problem solving 
in the topic area of environmental science. Students completed pre- and posttest concept 
maps, a relevant bookmarking measure, and a metacognitive survey. All process data was 
collected using computer trace data. During posttest map construction students had access 
to a simulated Web environment. Expert performance was collected and analyzed to infer 
the criteria used for scoring students' concept maps. 

Participants 

Sixty-nine middle school and high school students were randomly selected from 
three high schools, and one middle school to participate in the study. 

Computational Tools 

Three computational tools were designed to measure and report student’s 
information-based problem-solving behaviors (see Appendix A): (a) a Java-based Concept 
Mapper, (b) a simulated World Wide Web environment, (c) a Bookmarking applet. 
Performance Measures 

Problem solving performance . Our dependent measure of problem solving was 
students growth from pre-to-posttest concept mapping. The pretest concept mapping 
task was designed to measure students' content knowledge of environmental science by 
requiring them to construct semantic relationships among important concepts and facts. 

The posttest concept mapping task was completed in conjunction with a simulated Web 
environment that was designed to measure student problem solving. Four teachers and 
researchers choose important concepts and facts about the domain of environmental 
science. The final set of terms included: (1) atmosphere, (2) bacteria, (3) climate, 
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(4) carbon dioxide, (5) decomposition, (6) evaporation, (7) food chain, (8) oceans, 

(9) producers, (10) consumers, (11) respiration, (12) sunlight, (13) plants, 

(14) photosynthesis, (15) waste, (16) water cycle, (17) oxygen, and (18) 
greenhouse gases. 

The set of links used to represent the semantic relationships between the set of 
concepts and facts was based on prior research (Baker & Niemi, 1991; Baker, Niemi, 

& Herl, 1994; Baker, Niemi, Novak, & Herl, 1992; Herl et al., 1996; Jonassen, 1984; 
Lambiotte, Dansereau, Cross, & Reynolds, 1989), and included: (1) causes, 

(2) INFLUENCES, (3) PART OF, (4) PRODUCES, (5) REQUIRES, (6) USED FOR, AND (7) USES. 

Information seeking performance . The dependent variable that measured 
information seeking was relevant bookmarks. Relevant bookmarks told us how 
relevant the information students found was to helping them develop meaningful links 
between concepts. 

Process Measures 

Data was collected to evidence the affect of four process behaviors on final concept 
mapping performance and relevant bookmarking. Those process behaviors included 
exploratory processes, extraction processes, simplification processes and organizational 
processes. 

Exploratory Processes . All mouse clicks and pages visited during each student’s 
search were compiled and logged electronically. The log recorded the student ID number, 
the machine IP address, the student map ID, the time the student spent on each Web page, 
all the search terms the student used, each time the student accessed feedback, and all 
URLs the student visited. The log was used to determine student the student’s degrees of 
the exploratory information seeking processes of browsing, analytic, scan-and-select 
information seeking. Browsing behaviors were coded as the number of Web pages users 
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visit or revisit by clicking on hypertext links, clicking the Forward and Backward arrows 
of Netscape (1996), clicking the Reload button, or by selecting an item from the Go menu. 
One analytic search behavior was coded for every time a student entered a different search 
term or group of terms into the Web search engine. Scan-and-select behaviors were coded 
as each time the student returned to the retrieval list generated by an analytic search 
(Schacter et al., in press). 

Extraction processes . Relevant bookmarks measured how relevant the information 
students’ extracted and sent to specific concepts in their maps was. A relevant bookmark 
was a web document, a specific Universal Resource Locator (URL), that helped students 
make at least one meaningful link between two or more concepts. 

Simplification processes measured how well students simplified the task by 
attending to and focusing on which area(s) of their concept maps needed most 
improvement. These processes measured to what degree students sent bookmarks to 
concepts in their maps that needed most improvement. 

Organizational processes. Students’ organizational processes were measured 
through: (a) a metacognitive survey adapted from O’Neil and Abedi’s (1996) Self- 
Assessment Questionnaire, and (b) how often students accessed the feedback. 

Research generally confirms that learners are more effective when they attend to 
externally provided feedback (Bangert-Drowns, Kulik, Kulik & Morgan, 1991; Butler 
& Winne, 1995). At a global level, we report frequency counts of how many times 
throughtout the task the student accessed feedback about his or her performance. Each 
time a student accessed feedback, one point was awarded. 
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Procedures 

Students were trained how to use the computer-based concept mapping software 
through an interactive tutorial based on Herl (1995). All students, then were given 20 
minutes to individually construct a closed concept map about environmental science. Next 
students participated in a 12 minute tutorial about how to use the Bookmarking applet and 
the simulated Web environment in conjunction with the concept mapping software. 
Students were instructed to search the Web environment for 40 minutes to help them to 
improve their maps. 

Concept map scoring. Four junior high and elementary school science 
specialists were recruited to construct concept maps that were subsequently used as 
sources of expert criteria to score students' maps (Baker & Schacter, 1996; Herl, 1995; 
Herl, Niemi, & Baker, 1996). The semantic content score of a concept map is based 
on the semantic links constructed by experts in their concept maps. Based on Herl’s 
(1995) dissertation, two different methods of using expert maps to score student maps 
for semantic content were tested: (1) stringent, and (2) categorized. These two 
methods were created to compare the effects of removing subtle differences in the 
semantic meanings of the links. For each of these two methods, each expert's map was 
used to compute a total map score for each student, resulting in four scores for each 
student map. Using the stringent scoring method, 1 point was scored if a link in the 
student map matched the expert link exactly and 0 if it did not. The categorized method 
incorporated clustering of similar links. To compute categorized map scores links were 
categorized into causal (i.e., CAUSES, INFLUENCES) and conditional, (i.e., 
REQUIRES, USED FOR, PART OF) categories. The remaining links were 
unclassified. Using the categorized scoring method, 1 point was scored if the link in the 
student map matched the category of that link in the expert map and 0 if it did not. In 
either method, a student's link was scored 0 if the expert's map did not contain that 
particular link. The categorized method obviously generated map scores which were 
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greater than or equal to the scores using the stringent method. The number of links 
constructed in each expert's map represented the maximum possible score for each 
student map. The definition of semantic content score is the number of correct links in a 
student's map when using experts' maps as the sources of scoring criteria. 

Intermap agreement for the four combinations of expert maps ranged from 80% 
to 90%. Since these intermap agreements were high, the average of the four map 
scores was used to represent the score for each student map. The map score for each 
student is a weighted score, where the score for each link is proportional to the number 
of experts’ maps containing the same link. 

Relevant bookmark scoring . Student bookmarks (i.e., unique URLs sent to 
concepts) were scored against ratings of each Web page in the database for each concept. 
The Web page relevance rating rubric (Appendix B) was a five point scale based on 
previous research (Baker, Aschbacher, Niemi, & Sato, 1992). The 0-to-4 point scale 
assessed the relevance of each Web document for each of the 18 concepts. Rater agreement 
for Web documents for each concept were calculated. Cronbach alphas ranged from .74 to 
.93. Relevant documents were defined as documents that contained information that helped 
students meaningfully link two or more concepts. Relevant bookmarks served both as a 
dependent measure of information seeking performance, as well as a process measure of 
problem solving. 

As the information seeking performance dependent measure, the bookmarking 
score was represented in two ways (1) the ratio of relevant bookmarks over the total 
number of bookmarks sent, and (2) the total number of relevant bookmarks sent. 

As a problem solving process variable, we assessed whether or not the student 
found relevant documents for the concepts in which they needed most improvement. Ratio 
scores of (a) the number of bookmarks over the number of concepts that needed 
improvement, and (b) the number of relevant bookmarks over the number of concepts that 
needed improvement were calculated. 
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Metacognitive survey. The 32 item 4-point Likert scale survey ranged in scores 
from 32 to 128 and assessed the constructs of planning, cognitive strategy, monitoring, 
and searching strategies. Higher scores represented greater metacognitive ability. 
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Results 

Each hypotheses is discussed in the order it was introduced. Furthermore, as this 
was a feasibility study, we anticipated that because of the relatively new technologies we 
employed, that we might loose some student data due to computers crashing and data log 
management. For several of these hypotheses this was, in fact, the case. 

(1) With the addition of an electronic information space, students performance from 
pre-to posttest concept mapping scores will improve. 

Fifty-eight of the sixty-nine students completed both pre and posttest concept maps. 
A pairwise t-test confirmed our hypothesis that student performance improved with the 
addition of a Web environment (t (58) = 2.74, p = . 008). Means for posttest scores were 
(M = 4.56, SD = 4.01) greater than pretest scores (M = 3.75, SD = 2.88). 

(2) More exploratory information seeking behaviors ( i.e., browsing, scan and 
select, and analytic information seeking) will be predictive of higher scores on final concept 
mapping and higher scores on finding relevant information. 

Data was available for forty-eight of the fifty-eight students. A regression equation 
with final concept mapping scores as the dependent variable and browsing, scan and select, 
and analytic behaviors as the independent variables was run to test the first part of this 
hypothesis. None of these information seeking variables predicted scores on final concept 
mapping. To test the second part of this hypothesis, a regression equation with number of 
relevant bookmarks as the dependent variable and browsing, scan and select, and analytic 
behaviors as the independent variables was run. Again, no significant results were found. 
None of the information seeking variables predicted finding more relevant information. 
When a regression equation was run with total number of bookmarks as the dependent 
variable and browsing, scan and select, and analytic behaviors as the independent 
variables, these information seeking behaviors collectively predicted (F = 3.30075, 
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g = .03) finding more bookmarks. Further, browsing behaviors (t= 2.126, p = .041 1) 
significantly predicted finding more bookmarks. 

(3) Extracting more relevant information (i.e., bookmarking) from the Web 
environment will predict higher final concept mapping scores. 

Data was available for thirty-nine of the fifty-eight students to test this hypothesis. 

A regression analysis with concept mapping as the depending variables and total number of 
relevant bookmarks, percentage of relevant bookmarks and total number of bookmarks as 
the independent variables. Extracting more relevant information lead to better performance 
on final concept mapping (F = 5.87, p = .002). Both the number of relevant bookmarks (t 
= 3.232, p = .0026) and the percentage of relevant bookmarks (t = -2.279, 
p = .0287) predicted better final concept map performance. 

(4) High scores on metacognition will predict high scores on relevant bookmarks 
and final concept maps. 

Two regression equations were run to test this hypothesis. High metacognitive 
scores did not predict finding relevant bookmarks or scoring high on final concept maps. 
No subscales of the metacognitive survey predicted finding relevant bookmarks or scoring 
high on final concept maps. 

(5) High scores on task simplification (i.e., focusing on bookmarking concepts one 
knows little about) will predict higher final concept mapping scores. 

We tested this hypothesis on the sample of students who needed most improvement 
on their concept maps (n = 27). Students who needed most improvement were classified 
as those students that had incorrect links for more than ten concepts in their maps (i.e., 
over 50% of the concepts in their maps). A pairwise t -test was run comparing student 
final concept mapping scores to whether or not they focused on the concepts in their maps 
which needed most work. The results indicated that of the sample of students that needed 
most work on their maps those who focused on the concepts which needed most 
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improvement scored significantly higher than those student who did not (t = -2.58, p = 
.016). Means for final map performance were (M = 4.3 and M = 2. 1, respectively). 

(6) Students who organize and regulate their learning by accessing feedback on 
their concept mapping performance will score higher on final concept maps and finding 
relevant information than those who do not. 

To test this hypothesis we ran a one-way ANOVA by dividing use of feedback into 
three categories (i.e., never accessed feedback, accessed feedback 1 to 4 times, accessed 
feedback 5 times or more). We found no significant results for use of feedback 
contributing to either higher final concept mapping performance or a greater number of 
relevant bookmarks found. 
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Discussion 

This research study illustrated that a Web-based information rich environment can 
enhance student learning and problem solving. We argue that this finding is promising 
because it provides evidence that students can improve their understanding when working 
on realistic problems in realistic contexts that demand the activation of multiple cognitive 
processes. Even more promising is the feasibility of using networked computers to report 
on student problem solving processes that contributed to or detracted from student problem 
solving performance. 

Our analyses revealed that the problem solving processes of exploration, extraction, 
simplification and organization contributed to and predicted both information seeking and 
retrieval and problem solving performance. 

We found that more exploratory information seeking behaviors lead to finding more 
information, and that the information seeking behavior of browsing, in particular, predicted 
the finding of more information. As hypothesized the processes of extracting relevant 
information (i.e., relevant bookmarks) was critical to improving final problem solving 
performance as measured by the final concept mapping task. Being able to determine and 
extract relevant information contributed to students ability to construct more meaningful 
correct links. 

Student simplification processes also contributed to enhanced problem solving 
performance. We defined simplification processes as the student’s ability to focus his or 
her attention on a subsection of the task, rather than attack the whole complex problem 
without a plan. In our task, simplification involved students ability to attend to and focus 
on area(s) of their concept maps that needed most improvement. Both low performing and 
high performing students who focused on improving their knowledge of concepts they 
knew little about evidenced higher final concept mapping performance than students who 
did not focus on improving concepts where their knowledge was weak. 
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Finally, and surprisingly, organizational processes did not contribute to increased 
problem solving performance. We measured organizational processes in two ways: (a) 
how well students organized their own thinking, and (b) how well and how often students 
organized and used the external feedback they could receive from the system. We found 
that metacognitive ability (i.e., students organization of their own thinking) did not predict 
higher scores on either finding relevant information or higher scores on final concept 
mapping. We also found that using differential amounts of feedback from the system did 
not have any affects on final problem solving performance or on finding relevant 
information. These finding are surprising in light of both the metacognition and feedback 
research that show that metacognitive processes and using externally provided feedback 
amplify performance. 

To conclude, the results of this study indicate the feasibility of automated collection, 
scoring, and reporting of student complex problem solving process data and using that data 
to make inferences about problem solving performance. As we have shown, process data 
can be collected in inexpensive and non time intensive ways. Further, detailed reporting of 
students cognitive processes while engaged in learning activities is a feasible goal when 
using networked technologies. 

In our future research we need to correct system and technical errors which 
contributed to the loss of valuable process data. Further, future research should better 
define and test which cognitive processes under which types of task conditions are 
beneficial to student problem solving performance. Only then can claims calling for the 
link between assessment and instruction be realized. 
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Appendix A 
Computational Tools 

Java-Based Concept Mapp er. The concept mapper was programmed in Java 
and was accessed by using Netscape Navigator 3.0. The mapper interface consisted of 
three menus: (a) Session Menu, (b) Concept Menu, and (c) Link Menu. The Session 
Menu has save, erase, move, and link features. The Concept Menu consisted of 18 
concepts selected jointly by the research team and four junior high and elementary 
school science specialists. Concept terms could be selected and dragged and dropped 
onto the mapper grid. The Link Menu showed students the seven linking verbs 
determined from previous research (Herl, Niemi, & Baker, 1996) that students could 
use to connect concepts. Students could move concepts, link concepts, and erase 
concepts and links. Figure 1 illustrates the Java-Based Concept Mapper features. 

Simulated World Wide Web environment . The simulated WWW environment ran 
on an IBM dual 200 processor machine under Windows NT. The Web Server software 
was AOL. The Web environment consisted of over 90 Web documents, and over 1,000 
images and diagrams about environmental science and other topic areas. The database was 
made up of both relevant and irrelevant information about environmental science that varied 
in the degree of specificity, reading level, scope and comprehensiveness. Ninety percent of 
the information on the simulated Web was downloaded from the WWW using Web 
Whacker 1.0 software. The other 10% of the information was adapted from science 
textbooks and magazines. As on the Web, students were provided with three mechanisms 
to conduct searches: (a) a Search Engine, which supported keyword and Boolean 
searching, (b) a Directory, which had various topics organized in a hierarchical hypertext 
menu format, and (c) a Glossary, which housed over 70 terms related to environmental 
science (see Figure 2). Also, like the Web, the simulated environment contained 
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information that had a degree of inaccurate facts, misspellings, and ambiguous authors 
(Hernon, 1995). 

Bookmarking applet. The bookmarking applet was a Java tool that enabled 
students to send relevant Web pages they found during their search to concepts in the Java- 
based Concept Mapper. When students found Web pages that they believed supported the 
modifications they made to their concept maps, they were instructed to bookmark those 
pages and send them to a concept term(s) in the concept map. Students could also review 
the bookmarks they sent to concepts in their map by double-clicking on those concepts, 
which triggered a window that listed the Titles of the bookmarked pages. Figure 3 is a 
series of screen shots showing how the bookmarking applet works. 
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Appendix B 

Web Page Relevance Rating Rubric 



The concept or term does not appear in the document. 

The concept or term appears in the document, but it is not defined, explained, or 
linked to other concepts within the document. 

The concept or term appears in the document and is defined at a global general 
level. The concept or term is not explicitly linked to other concepts or terms. 

The concept or term appears in the document, is defined, and is also explicitly 
linked to another concept or term. 

The concept or terms appear in the document, is defined, and is explicitly linked to 
more than one concept or terms. 
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